Amazon SageMaker Studio can help you build, train, debug, deploy, and monitor your models and manage your machine learning (ML) workflows. Amazon SageMaker Pipelines enables you to build a secure, scalable, and flexible MLOps platform within Studio. In this post, we explain how to run PySpark processing jobs within a pipeline. This enables anyone that wants to train a model using Pipelines to also preprocess training data, postprocess inference data, or evaluate models using PySpark. This capability is especially relevant when you need to process large-scale data.

Welcome back to my blog, where I share insights and tips on machine learning workflows using Sagemaker Pipelines. If you're new here, I recommend checking out my first post to learn more about this AWS fully managed machine learning service. In my second post, I discussed how parameterization can help you customize the workflow and make it more flexible and efficient. After using Sagemaker Pipelines extensively in real-life projects, I've gained a comprehensive understanding of the service. In this post, I'll summarize the key benefits of using Sagemaker Pipelines and the limitations you should consider before implementing it. This service is integrated with Sagemaker directly, so the user doesn't have to deal with other AWS services.

SageMaker is a fully managed machine learning service on the AWS cloud. The motivation behind this platform is to make it easy to build robust machine learning pipelines on top of managed AWS cloud services. Unfortunately, the abstractions that lead to its simplicity make it quite difficult to customize. This article will explain how you can inject your custom training and inference code into a prebuilt SageMaker pipeline. Our main goal is to enable Intel AI Analytics Toolkit accelerated software in SageMaker pipelines.

This article was originally an episode of MLOps Live, an interactive Q&A session where ML practitioners answer questions from other ML practitioners. Every episode is focused on one specific ML topic, and during this one, we talked to Mateusz Opala about leveraging unlabeled image data with self-supervised learning or pseudo-labeling. But, if you prefer a written version, here it is! Sabine: With us today, we have Mateusz Opala, who is going to be answering questions about leveraging unlabeled image data with self-supervised learning or pseudo-labeling. Sabine: It's great to have you. Mateusz has held a number of leading machine learning positions at companies like Netguru and Brainly. So, Mateusz, you have a background in computer science, but how did you get more into the machine learning side of things? Mateusz: It all started during my sophomore year at university. One of my professors told me that Andrew Ng was doing his first iteration of the famous course on machine learning on Coursera. I kind of started from there, then did a bachelor thesis on deep unsupervised learning and went to Siemens to work in deep learning, and then all my positions were strictly about machine learning. Sabine: You've been on that path ever since? I worked for some time before as a backend engineer. But for most of the time in my career, I was a machine learning engineer/data scientist. Sabine: Mateusz, to warm you up.

Originally published on Towards AI the World's Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider...

One of the main challenges in a machine learning (ML) project implementation is the variety and high number of development artifacts and tools used. This includes code in notebooks, modules for data processing and transformation, environment configuration, inference pipeline, and orchestration code. In production workloads, the ML model created within your development framework is almost never the end of the work, but is a part of a larger application or workflow. Another challenge is the varied nature of ML development activities performed by different user roles. For example, the DevOps engineer develops infrastructure components, such as CI/CD automation, builds production inference pipelines, and configures security and networking.

Cloud computing is one of the fastest growing skills in the Machine Learning world. Among cloud services companies, Amazon stands out for providing one of the most advanced tools for Machine Learning: Amazon SageMaker. Using SageMaker you can, among many other things, build, test and deploy Machine Learning models. Furthermore, you can create End-to-End pipelines in order to integrate your models in a CI/CD environment. In this post we are going to use Amazon SageMaker to create an End-to-End pipeline step by step.

The process of extracting, cleaning, manipulating, and encoding data from raw sources and preparing it to be consumed by machine learning (ML) algorithms is an important, expensive, and time-consuming part of data science. Managing these data pipelines for either training or inference is a challenge for data science teams, however, and can take valuable time away that could be better used towards experimenting with new features or optimizing model performance with different algorithms or hyperparameter tuning. Many ML use cases such as churn prediction, fraud detection, or predictive maintenance rely on models trained from historical datasets that build up over time. The set of feature engineering steps a data scientist defined and performed on historical data for one time period needs to be applied towards any new data after that period, as models trained from historic features need to make predictions on features derived from the new data. Instead of manually performing these feature transformations on new data as it arrives, data scientists can create a data preprocessing pipeline to perform the desired set of feature engineering steps that runs automatically whenever new raw data is available.

Amazon today launched SageMaker Data Wrangler, a new AWS service designed to speed up data prep for machine learning and AI applications. Alongside it, the company took the wraps off of SageMaker Feature Store, a purpose-built product for naming, organizing, finding, and sharing features, or the individual independent variables that act as inputs in a machine learning system. Beyond this, Amazon unveiled SageMaker Pipelines, which CEO Andy Jassy described as a CI/CD service for AI. And the company detailed DevOps Guru and QuickSight Q, offerings that uses machine learning to identify operational issues, provide business intelligence, and find answers to questions in knowledge stores, as well as new products on the contact center and industrial sides of Amazon's business. During a keynote at Amazon's re:Invent conference, Jassy said that Data Wrangler has over 300 built-in conversion transformation types.

Amazon SageMaker Pipelines is the first purpose-built, easy-to-use continuous integration and continuous delivery (CI/CD) service for machine learning. With SageMaker Pipelines, you can create, automate, and manage end-to-end ML workflows at scale. Since it is purpose-built for machine learning, SageMaker Pipelines helps you automate different steps of the ML workflow, including data loading, data transformation, training and tuning, and deployment. With SageMaker Pipelines, you can build dozens of ML models a week, manage massive volumes of data, thousands of training experiments, and hundreds of different model versions. You can share and re-use workflows to recreate or optimize models, helping you scale ML throughout your organization.